Crowdsourcing a High-Risk Classifier for Multiple Myeloma Patients

Background

Multiple myeloma (MM) is a cancer of the plasma cells, and its clinical course depends on a complex interplay of clinical traits and molecular characteristics of the tumor. While current therapeutic combinations work well for a majority of patients, a subset of MM patients still rapidly progress or die within 18 months after diagnosis. Thus, there is an urgent need for a precise risk stratification model to identify these patients to allow selection of alternative therapeutic approaches. Current risk stratification practice is based on plasma cell cytogenetics and clinical stage at presentation, and, to a lesser degree, myeloma cell gene expression. The Myeloma Genome Project (MGP), an industry-academia collaborative project aims to develop highly accurate, clinically-implementable prognostic tests for MM patients. As part of the broader MGP effort, The MM DREAM (Dialogue for Reverse Engineering Assessments and Methods) challenge was conceived as an innovative approach to accelerate the development and evaluation of risk models in newly diagnosed MM. We collected large and diverse datasets from multiple collaborative groups and recruited a community of computational biologists, statisticians and leading experts in MM to build and test models (including the DREAM community of 20,000 solvers).

Methods

We assembled nine datasets comprised of 3,077 subjects with whole exome profiling (N = 1,273), gene expression profiling (N = 2,431), and clinical and cytogenetic data. Four data sets are publicly available and were provided for model training/development, while 5 data sets are previously unpublished and blinded to participants for model validation. Unpublished data were generously provided by an array of public, private and non-profit contributors including the University of Arkansas for Medical Sciences and Dana Farber Cancer Institute on behalf of the MGP, the Multiple Myeloma Research Foundation's Personalized Medicine Initiative (www.themmrf.org), Moffitt Cancer Center and M2Gen, and the University of Heidelberg. The Challenge goal was to accurately predict which newly diagnosed MM patients would progress (or die) within 18 months of diagnosis. We challenged the community to identify high-risk patients using DNA-based, RNA-based, or a combination of RNA, DNA, cytogenetic and demographic features. As an additional incentive, $150,000 in prizes will be awarded to top performing teams. By the first month of the competition, 290 teams from around the world had signed up for the challenge. We compare competitors' performance against state of the art classifiers based on gene expression (e.g. UAMS-70, EMC-92), patient characteristics (e.g. International Staging System stage combined with age), and cytogenetics. These were used as a benchmark upon which winning models needed to improve. The challenge was hosted on Sage Bionetworks' Synapse platform, which enabled development, submission and evaluation of models using an automated framework. A Docker framework was used to ensure portability and to allow secure use of unpublished validation data. Participants were evaluated on their ability to predict high risk patients using a variety of metrics including integrated AUC, AUC and precision-recall AUC.

Results / Discussion

Based on implementation of several published benchmark models on a preliminary set of validation data, the top performing published DNA-based model was based on mutation burden (Miller et al., AUC = 0.70, iAUC = 0.71), the top performing published RNA-based model was the UAMS-17 (Shaughnessy et al., AUC = 0.52), and the top performing clinical based model was a combination of age and ISS (AUC = 0.69, iAUC=0.68). The unique combination of over 3,000 patients' worth of molecular and clinical data allowed precise model generation while the collection of multiple distinct (unpublished) validation cohorts prevented model over fitting. Top performing teams identified novel prognostic feature sets for high risk MM, establishing new benchmarks for the field, and final results from the top performing models will be presented. Such novel models will be integrated with the broader efforts within MGP for validation and clinical test development. The results of this effort demonstrate that data sharing coupled with the power of crowdsourcing can contribute to the development of robust and clinically-implementable models for risk-stratification of patients with MM.